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Abstract 

The  U.S.  Government  has  looked  for  effective  ways  of  reducing  acquisition  cost 
and  schedule  overruns  for  decades.  The  task  of  isolating  the  root  cause  of  these  overruns 
has  been  difficult.  Consequently,  it  has  been  difficult  for  the  Government  to  create 
effective  policies  that  prevent  overruns  from  recurring.  In  1998,  the  Government 
Accountability  Office  (GAO)  undertook  this  problem,  and  looked  to  successful  DOD  and 
commercial  companies  for  solutions.  They  found  that  using  mature  technology,  having 
complete  product  designs,  and  having  production  processes  under  control  was  critical  to 
successfully  developing  new  products.  The  GAO  combined  these  concepts  into  a  single 
acquisition  practice  that  they  call  a  Knowledge-Based  Approach.  They  postulate  that 
programs  that  adhere  to  the  Knowledge-Based  Approach  will  experience  better  program 
outcomes  than  programs  that  do  not.  This  thesis  validates  the  GAO’s  claim  by 
comparing  the  outcomes  of  programs  that  met  the  Knowledge-Based  Approach  criteria 
with  those  that  did  not.  Our  findings  suggest  that  the  GAO’s  claim  is  accurate.  While 
their  approach  may  not  be  a  single  means  for  success,  programs  that  employed  their 
approach  generally  performed  better.  The  programs  that  met  the  GAO’s  criteria 
experienced  a  smaller  variation  of  outcomes  and  appeared  less  likely  to  spiral  out  of 
control. 


IV 


AFIT/GC  A/ENV/ 1 0-M05 


/  dedicate  this  work  to 
me 


my  wife  and  daughter.  Their  support  and  sacrifice  have  cdlowed 
to  complete  this  study  on  time  and  on  budget. 


v 


Acknowledgements 


First,  I  would  like  to  thank  my  wife  and  daughter.  Their  love,  support,  and 
sacrifice  have  afforded  me  the  opportunity  to  focus  on  this  research,  they  have  given  me 
the  opportunity  to  garner  many  wonderful  experiences  at  AFIT. 

I  would  also  like  to  extend  my  sincere  gratitude  to  my  thesis  committee.  The 
constant  encouragement  and  guidance  from  my  advisor,  Lt  Col  R.  David  Fass,  was 
critical  in  completing  this  research.  I  am  grateful  for  the  countless  hours  Lt.  Col  Fass 
dedicated  to  supporting  this  endeavor  and  to  helping  me  become  a  better  researcher.  I  am 
grateful  to  Lt.  Col  Eric  Unger  for  providing  me  with  the  education  and  support  I  needed 
to  complete  this  thesis,  as  well  as  giving  me  the  tools  I  will  need  to  become  a  successful 
analyst.  Finally  yet  importantly,  I  am  grateful  for  the  time  and  statistical  expertise  that 
Capt  Patrick  Chapin  contributed  to  my  research.  Capt  Chapin  played  a  pivotal  role  in 
determining  the  most  appropriate  method  for  testing  the  KAT;  his  insight  was  invaluable. 
It  has  been  a  privilege  to  work  with  each  committee  member,  and  I  believe  I  am  a  better 
researcher  for  having  done  so. 

In  addition,  I  would  like  to  thank  Maj  Matthew  Laubacher.  His  sponsorship  and 
mentoring  have  made  this  possible.  He  was  influential  in  my  decision  to  pursue  this 
undertaking  and  a  significant  support  throughout  the  course  of  it. 

Finally,  I  would  like  to  thank  my  classmates,  each  of  them  have  been  instrumental 
to  my  successes  at  AFIT.  They  have  made  this  an  incredible  experience. 


Dan  Wyman 


Table  of  Contents 


Page 


Abstract . iv 

Dedication . v 

Acknowledgements . vi 

Table  of  Figures . ix 

List  of  Tables . x 

I:  Introduction . 11 

Introduction . 11 

The  Current  Acquisition  Movement . 13 

The  GAO  Studied  Best  Practices . 13 

Follow-up  Assessments . 14 

Research  Question: . 15 

II:  Literature  Review . 18 

Introduction . 18 

Background . 19 

Post  1986  Packard  Commission  Reform . 21 

GAO’s  KAT . 22 

DOD  versus  Commercial  Industry . 23 

GAO’s  Knowledge-Based  Acquisition  Theory . 25 

Knowledge  Point  1:  Mature  Technology . 27 

Knowledge  Point  2:  Product  Design . 29 

Knowledge  Point  3:  Production  Maturity . 30 

Current  Acquisition  Reform . 32 

Weapon  System  Reform  Act  of  2009 . 33 

Chapter  III:  Data  Collection  and  Methodology . 36 

Data  Source  and  Variables . 36 

Limitations  of  the  Data . 38 

Dependent  Variables . 40 

Dependent  Variable  Description . 41 

Methodology . 44 

vii 


Page 


Distribution  of  Variables . 46 

Controlling  for  Program  Age . 46 

Chapter  IV:  Analysis  and  Results . 48 

Test  of  Hypotheses . 48 

Decreased  Variability . 52 

Controlling  for  Program  Age . 52 

Chapter  V:  Conclusions . 60 

Limitations . 61 

Impact  to  the  Acquisition  Community . 62 

Conclusion . 63 

Appendix  A:  Technology  Readiness  Level’s . 64 

(GAO,  2009,  pp.  174-175) . 64 

Appendix  B:  Summary  of  KP  achievement  by  Program  Type  and  Dependent  Variable65 
Works  Cited . 68 


viii 


Table  of  Figures 


Page 

Figure  1:  Knowledge  Assessment  for  Weapon  System  Programs  in  2008* .  16 

Figure  2:  Achieved  Technology  Development . 26 

Figure  3:  Knowledge  Point  Schedule . 31 

Figure  4:  Knowledge  Point  Attainment  (GAO  2008) . 37 

Figure  5:  Data  Deduction  (GAO,  2008,  p.  27) . 39 

Figure  6:  Program  Performance  with  Respect  to  Program  Baseline  (GAO,  2008) . 41 

Figure  7:  Unit  Cost  Function . 50 

Figure  8:  KP1  versus  DV's . 55 

Figure  9:  KP2  versus  DV's . 56 

Figure  10:  KP3  versus  DV's . 57 

Figure  11:  KP  accomplishment  by  program  type . 58 


IX 


List  of  Tables 


Page 


Table  1:  Total  Cost  of  DOD's  Top  5  Programs  (FY06  Dollars) . 32 

Table  2:  Data  Demographics . 40 

Table  3:  Variable  List . 45 

Table  4:  Descriptive  Statistics  (one-tailed) . 49 

Table  5:  Results  of  Analyses . 51 

Table  6:  Results  of  Analysis,  Controlling  for  Age . 53 


x 


BEST  PRACTICES  IN  GOVERNMENT  ACQUISITION: 

A  TEST  OF  THE  GOVERNMENT  ACCOUNTABILITY  OFFICE’S  KNOWLEDGE- 

BASED  ACQUISITION  THEORY 


I:  Introduction 


Introduction 

The  U.S.  Government  has  searched  for  effective  ways  of  reducing  its  acquisition 
cost  and  schedule  overruns  for  decades  (Deloitte  Consulting  LLP,  2008;  Hanks, 
Axelband,  Lindsay,  Malik,  &  Steele,  2005;  GAO,  2002).  Acquisition  reform  efforts  have 
been  around  for  decades,  yet  acquisition  program  outcomes  have  improved  very  little 
(Friedman,  2009;  Hanks,  Axelband,  Lindsay,  Malik,  &  Steele,  2005).  The 
ineffectiveness  of  current  acquisition  practices  is  a  challenge,  and  has  limited  the  DOD’s 
ability  to  supply  the  warfighter  with  systems  on  time  and  on  cost  (Shimel,  2008).  Poor 
acquisition  program  outcomes  have  become  notorious  throughout  the  DOD,  as  they  have 
become  a  hindrance  to  the  acquisition  customer,  as  well  as  to  internal  and  external 
acquisition  stakeholders.  Deloitte  Consulting  conducted  a  survey  of  government  and 
Aerospace  &  Defense  executives  regarding  program  execution  problems.  Deloitte  found 
that  over  43%  of  government  and  Aerospace  &  Defense  executives  polled  thought  that 
program  execution  problems  were  as  serious  as  the  current  housing  and  banking  crisis 
(Deloitte  Consulting  LLP,  2008,  p.  1).  Additionally,  because  of  Congress’s  concern  with 
program  performance,  Congress  has  established  mandatory  reporting  requirements  for 
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those  programs  in  which  unit  cost  has  exceeded  15  percent  of  the  originally  planned  cost 
(Shimel,  2008). 

The  DOD  acquisition  portfolio  is  enormous,  containing  96  major  programs  valued 
at  approximately  $1.6  Trillion  in  2008.  Further,  the  cost  growth  of  these  programs  in 
2008  was  valued  at  $296  Billion  and  the  average  program  was  22  months  behind 
schedule  (GAO,  2009,  p.  7).  On  average,  cost  and  schedule  overruns  have  been 
increasing  by  roughly  1.86  percent  annually  and  it  has  been  estimated  that  they  will 
increase  from  the  already  high  rate  of  25  percent  in  2009,  to  over  46  percent  by  2018 
(Deloitte  Consulting  LLP,  2008,  p.  2;  GAO,  2009). 

In  2009,  the  VH-71  program  experienced  a  unit  cost  growth  of  fifty  percent  from 
its  estimate  in  2006,  and  it  cut  its  production  quantity  in  half  (Fein,  2009;  GAO,  2009). 
The  VH-71  program’s  overrun  was  a  Nunn-McCurdy  Breach,  wherein  the  Secretary  of 
Defense  must  certify  that  the  program  meets  several  requirements  in  order  to  keep  the 
program  “alive”  (BNA  Federal  Contracts  Daily,  2007).  The  user  will  receive  half  the 
units  they  expected  and  will  pay  more  for  them.  This  is  not  atypical,  programs  like  the 
C-130  Avionics  program,  the  Expeditionary  Fighting  Vehicle,  the  Joint  Air-to-Surface 
Standoff  Missile,  the  Land  Warrior,  and  the  Warfighter  Information  Network  all  suffered 
Nunn-McCurdy  breaches  due  to  unit  cost  growth  of  more  than  25%  of  their  current 
baseline  or  50%  over  the  Approved  Program  Baseline  (BNA  Federal  Contracts  Daily, 
2007).  Clearly,  a  need  exists  for  more  effective  acquisition  practices  to  improve  this 
trend  and  help  programs  stay  within  cost  and  schedule  estimates. 
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The  Current  Acquisition  Movement 


The  Office  of  the  Assistant  Secretary  of  the  Air  Force  published  the  Acquisition 
Improvement  Plan  in  2009.  Their  focus  was  on  recapturing  acquisition  excellence  by 
establishing  an  acquisition  framework  that  instilled  rigor,  reliability,  and  transparency  in 
the  acquisition  system.  The  Acquisition  Improvement  Plan  required  the  use  of 
incremental  acquisition  strategies  that  reduced  cost,  schedule,  and  technical  risk,  while 
producing  operational  capabilities  early  (Office  of  the  Assistant  Secretary  of  the  Air 
Force,  2009).  Lastly,  the  plan  sought  to  “implement  means  to  increase  cost  estimating 
confidence  levels  and  establish  more  realistic  program  budgets  (Office  of  the  Assistant 
Secretary  of  the  Air  Force,  2009).”  We  suspect  that  the  Government  Accountability 
Office’s  (GAO)  Knowledge-Based  Approach  was  influential  in  the  creation  of  these 
goals.  The  GAO’s  Knowledge-Based  Approach  centers  on  the  concept  that  programs  can 
achieve  better  results  by  reducing  their  technology  risk  early  in  the  program  lifecycle. 
Thus,  the  GAO’s  approach  should  be  an  effective  way  for  programs  to  achieve  the 
challenging  demands  of  the  Acquisition  Improvement  Plan. 

The  GAO  Studied  Best  Practices 

The  GAO  conducted  an  in-depth  study  of  the  best  practices  used  by  DOD  and 
Commercial  industry  and  then  used  their  findings  to  develop  the  Knowledge-Based 
Approach.  Their  goal  was  to  help  program  managers  achieve  better  program  outcomes 
by  leveraging  the  acquisition  community’s  best  practices  (GAO,  1998).  The  GAO 
acquired  these  best  practices  through  interviews  of  what  they  considered  the  best  DOD 
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and  commercial  companies  with  regard  to  developing  and  manufacturing  new  products 
(GAO,  1998).  The  GAO  found  that  an  important  characteristic  of  successful  commercial 
and  DOD  organizations  was  that  they  had  attained  essential  knowledge  of  their  programs 
at  or  before  critical  points  in  their  new-product  development  processes.  These  programs 
had  mature  technology  prior  to  program  initiation,  they  had  complete  product  designs 
prior  to  their  product  design  review,  and  they  had  their  production  processes  under 
control  before  beginning  production. 

Consequently,  the  GAO  recommended  that  the  DOD  adopt  these  practices 
through  an  approach  the  GAO  calls  the  Knowledge-Based  Approach.  The  GAO  believes 
that  this  approach  will  help  DOD  programs  achieve  better  program  outcomes  based  on 
the  GAO’s  extensive  study  of  new-product  development  best  practices  (GAO,  1998; 
GAO,  2002).  This  approach  has  been  instrumental  to  the  success  of  commercial  firms 
like  Boeing,  Chrysler  Corporation,  Cummins  Engine  Company,  Ford  Motor  Company, 
Honda  Motor  Company,  and  Hughes  Space  and  Communications,  all  considered  among 
the  best  organizations  in  developing  new  products  (GAO,  1998,  p.  3). 

Follow-up  Assessments 

The  GAO  collected  program  performance  data  on  major  acquisition  programs  as  a 
follow-up  to  their  original  study.  Each  program  was  chosen  because  of  its  high  dollar 
value,  acquisition  stage,  or  because  it  had  attracted  congressional  interest  (GAO,  2008). 
The  data  sources  included  the  Selected  Acquisition  Reports  and  the  Defense  Acquisition 
Management  Information  Retrieval  Purview  system,  which  provided  the  financial, 
schedule,  and  quantity  information  for  each  program.  Additionally,  the  GAO  used 
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questionnaires  and  interviews  to  gather  information  about  program  office  staffing, 
program  requirement  changes,  and  other  program  aspects  (GAO,  2009).  The  GAO  has 
presented  their  audit  findings  annually,  beginning  in  2003,  in  GAO  reports  titled 
Assessments  of  Selected  Weapon  Programs.  As  we  will  discuss,  these  reports  were  the 
basis  of  our  analyses. 

Purpose  of  This  Study 

The  objective  of  our  study  was  to  test  the  GAO’s  Knowledge-Based  Approach 
(hereafter  termed  the  Knowledge-Based  Acquisition  Theory  (KAT)),  to  determine  if 
programs  that  met  the  KAT  criteria  performed  better  than  those  that  did  not.  By  testing 
the  KAT,  we  felt  we  would  better  understand  the  relationship  between  the  KAT  and 
program  performance,  thus  giving  additional  credence  to  the  GAO’s  methodology.  Our 
test  focused  on  the  following  research  question: 

Research  Question: 

Do  defense  acquisition  programs  that  adhere  to  the  GAO’s  Knowledge-Based 

Approach  perform*  better  than  programs  that  do  not? 

* Performance  measured  using  cost,  schedule  and  quantity  variables. 

This  study  will  contribute  to  the  “Acquisition  Community  of  Practice”  (Defense 
Acquisition  University).  Our  results  will  provide  program  managers  with  a  better 
understanding  of  the  implications  of  their  decisions  regarding  the  GAO’s  Knowledge- 
Based  Approach.  Furthermore,  isolation  of  the  factors  most  critical  to  program  success 
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will  help  decision  makers  better  understand  the  trade  space  surrounding  their  decisions, 


as  well  as  better  understand  the  potential  consequences  of  their  decisions. 
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programs 


0  percent  of 
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Figure  1:  Knowledge  Assessment  for  Weapon  System  Programs  in  2008 


(GAO,  2008) 


The  GAO’s  Knowledge-Based  Approach  seems  to  be  a  logical  approach  to 
obtaining  better  program  results.  However,  in  2008,  the  GAO  reported  that  none  of  the 
programs  they  assessed  at  Milestone  C  (Production  Start)  had  achieved  the  respective 
Knowledge-Based  Approach  criteria,  and  that  88  percent  of  programs  were  initiated 
(Development  Start)  without  mature  technology  (Figure  1).  We  anticipate  that  our 
findings  will  confirm  the  GAO’s  claim  that  programs  that  adhere  to  the  Knowledge- 
Based  Approach  will  experience  better  outcomes,  and  that  our  findings  will  encourage 
program  managers  to  adopt  the  GAO’s  Knowledge-Based  Approach. 

We  organized  the  remainder  of  this  document  as  follows:  Chapter  II  provides  an 
in  depth  look  at  the  Knowledge-Based  Acquisition  Theory  as  well  as  a  historical  glance 
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at  previous  acquisition  reform  efforts.  In  Chapter  III,  we  explain  our  data  source  and  the 
variables  we  used  for  our  analysis.  We  also  explain  the  methods  we  used  to  test  our 
hypotheses.  In  Chapter  IV,  we  describe  the  results  of  our  analysis.  Lastly,  in  Chapter  V 
we  summarize  the  results  and  suggest  the  policy  and  procedure  implications  of  our 
results. 
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II:  Literature  Review 


Introduction 

Over  the  past  twenty-five  years,  acquisition  reform  initiatives  have  focused  on 
ways  to  improve  the  efficiency  and  effectiveness  of  acquisitions,  so  that  the  Department 
of  Defense  (DOD)  can  reduce  cost  and  schedule  overruns  (Hanks,  Axelband,  Lindsay, 
Malik,  &  Steele,  2005;  Christensen,  Searle,  &  Vickery,  1999).  The  GAO  has  offered  its 
Knowledge-Based  Approach  as  an  effective  way  for  the  DOD  to  improve  acquisition 
program  outcomes,  based  on  the  results  of  GAO’s  study  of  best  practices  within  DOD 
and  industry  (GAO,  1998).  Although  the  GAO’s  approach  has  influenced  the 
Acquisition  Improvement  Plan,  the  GAO’s  approach  has  not  been  readily  adopted,  and 
the  management  and  execution  problems  have  become  "too  big  to  ignore”  (Deloitte 
Consulting  LLP,  2008). 

Department  of  Defense  (DOD)  acquisition  programs  continue  to  have  cost  and 
schedule  overruns  (GAO,  2002,  p.  2;  Deloitte  Consulting  LLP,  2008;  Schwartz,  2009). 
These  overruns  reduce  the  DOD’s  buying  power  and  reduce  the  funding  available  for 
other  DOD  priorities  (GAO,  2008,  p.  5;  Gansler,  1989,  p.  170).  Overruns  are  not  new  to 
DOD  acquisitions.  However,  overruns  are  often  viewed  as  singular  events,  rather  than 
systemic  ones  (Shimel,  2008).  Further,  cost  and  schedule  growth  has  invoked  concern 
from  both  Congress  and  the  American  public  in  the  DOD’s  ability  to  meet  the  urgent  and 
growing  needs  of  the  warfighter  (GAO,  2005;  Schwartz,  2009).  In  2009,  the  DOD 
acquisition  portfolio  was  valued  at  $1.6  Trillion,  and  the  portfolio  had  accumulated  an 
estimated  $296  Billion  in  program  cost  growth  (GAO,  2009).  This  cost  growth  is 
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equivalent  to  the  pay  and  expenses  of  all  members  of  the  armed  services  for  two  years 


(1 1 1-23,  2009,  p.  12).  These  persistent  cost  and  schedule  overruns  make  it  hard  for  the 
DOD  to  accurately  forecast  its  financial  requirements  and  have  been  the  catalyst 
motivating  the  DOD  and  Congress  to  pursue  more  efficient  acquisition  practices.  (GAO, 
2009,  p.  9) 


Background 

Congress  and  the  DOD  have  looked  for  ways  to  improve  the  acquisition  process 
for  decades  (Christensen,  Searle,  &  Vickery,  1999;  Hanks,  Axelband,  Lindsay,  Malik,  & 
Steele,  2005;  The  President's  Blue  Ribbon  Commission  on  Defense  Management,  1986). 
Each  increment  of  change  that  has  been  implemented  has  attempted  to  make  DOD 
acquisitions  a  more  efficient  and  effective  process.  However,  the  reform  initiatives  from 
the  1960s  to  the  1990s  brought  little  improvement  to  the  effectiveness  of  the  system  as 
indicated  by  a  consistent  cost  growth  of  approximately  twenty  percent  throughout  that 
period  (Christensen,  Searle,  &  Vickery,  1999,  p.  4).  During  the  1980’s,  reform  initiatives 
focused  on  fraud,  waste,  and  abuse.  However,  the  focus  changed  in  the  late  1980’s  when 
it  became  apparent  that  the  cost  of  fraud,  waste,  and  abuse  was  inconsequential  compared 
to  the  cost  of  the  heavily  legislated  procurement  process  (Hanks,  Axelband,  Lindsay, 
Malik,  &  Steele,  2005). 

Consequently,  in  1986  President  Reagan  initiated  significant  changes  to  the 
acquisition  process.  At  that  time,  the  overall  sentiment  toward  DOD  acquisitions  was 
that  the  DOD  needed  to  become  more  “responsive,  effective,  and  efficient”  in  its 
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acquisition  practices  (Hanks,  Axelband,  Lindsay,  Malik,  &  Steele,  2005).  Consequently, 
President  Reagan  appointed  Mr.  David  Packard  to  direct  the  “Presidents  Blue  Ribbon 
Commission  on  Defense  Management”  in  1985.  The  Packard  Commission  spent  a  year 
studying  ways  in  which  the  DOD  could  improve  its  management  through  an  in  depth 
look  at  both  government  and  commercial  companies  that  produced  similar  products  to 
those  produced  in  the  DOD.  The  commission  focused  on  the  underlying  business  models 
of  successful  companies  to  see  what  models  they  could  use  as  a  foundation  for  an 
improved  DOD  acquisition  model.  As  we  will  discuss  later,  the  approach  taken  by  the 
GAO  in  1998  was  very  similar  to  the  approach  taken  by  the  Packard  Commission  in 
1986. 

The  Packard  Commission  found  that  the  DOD  needed  to  reduce  the  “red  tape” 
that  was  restricting  its  employees  from  being  efficient.  They  also  found  that  the  current 
process  was  “inflexible,”  and  that  the  legislation  was  often  the  cause  of  the  waste  it  aimed 
to  eliminate  (President’s  Blue  Ribbon  Commission  on  Defense  Management,  1986).  As 
a  result,  their  efforts  aimed  at  reducing  the  “bureaucratic  inefficiencies”  found  in  the 
system  by  consolidating  acquisition  legislation  and  streamlining  the  process  (President’s 
Blue  Ribbon  Commission  on  Defense  Management,  1986). 

In  addition  to  finding  ways  to  streamline  the  process,  the  1986  Packard 
Commission  examined  how  technology  push  and  user  pull  could  have  specific 
implications  on  government  acquisitions  (The  President's  Blue  Ribbon  Commission  on 
Defense  Management,  1986).  As  they  described,  a  user  pull  situation  occurs  when  users 
assess  their  current  capabilities  and  then  request  new  assets  to  fill  any  capability 
deficiencies  (President’s  Blue  Ribbon  Commission  on  Defense  Management,  1986).  It  is 
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an  effective  approach  in  terms  of  keeping  the  requirements  generation  process 


requirements  driven  since  the  users  are  generating  requirements  based  on  current  needs, 
and  not  on  a  desire  to  have  a  “new”  technology.  Unfortunately,  the  typical  user  is 
unfamiliar  with  the  technology  development  process  and  usually  has  only  a  limited 
understanding  of  the  implications  of  their  “requests”  (President’s  Blue  Ribbon 
Commission  on  Defense  Management,  1986).  A  technology  push  situation  occurs  when 
new  technologies  are  “sold”  to  users.  Users  then  generate  requirements  based  around  the 
new  concepts.  As  we  will  discuss  later,  the  GAO’s  KAT  recommends  that  new  product 
development  requests  originate  from  user-defined  (“user  pulled”)  capability  gaps  and  that 
the  acquisition  community  fill  gaps  using  existing,  mature,  technologies  (GAO,  1998,  pp. 
24-25) 


Post  1986  Packard  Commission  Reform 

For  more  than  a  decade  following  the  1986  Packard  commission,  acquisition 
reform  focused  on  ways  to  achieve  quicker,  more  efficient  processes.  Throughout  the 
1990’s  the  DOD  implemented  over  sixty  acquisition  reform  initiatives  focusing  on 
“faster,  better,  and  cheaper”  acquisitions  (Hanks,  Axelband,  Lindsay,  Malik,  &  Steele, 
2005)  .  These  initiatives  focused  on  helping  the  DOD  expedite  its  acquisition  processes 
and  allow  it  to  take  an  aggressive  approach  to  new  product  development.  These  changes 
included  streamlining  the  acquisition  hierarchy,  applying  “off-the-shelf’  components  to 
new  products,  and  using  cost  and  performance  tradeoff  studies  when  determining 
program  requirements  (Templin  &  Christensen).  However,  the  reform  initiatives  in  the 
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1 990’s  also  led  to  a  significant  reduction  in  the  size  of  the  government  acquisition 
workforce,  and  in  turn  led  to  an  increase  in  outsourcing  (Thomas,  2008). 

After  2001,  the  reform  focused  primarily  on  improving  accountability  and 
oversight  (Hanks,  Axelband,  Lindsay,  Malik,  &  Steele,  2005).  In  addition,  current 
reform  has  placed  the  burden  of  program  performance  on  the  program  manager  (Hanks, 
Axelband,  Lindsay,  Malik,  &  Steele,  2005,  p.  17).  Managing  a  program  is  a  daunting 
task  during  a  time  when  much  of  the  program  execution  responsibility  rests  with  the 
major  defense  contractors  (Deloitte  Consulting  LLP,  2008,  p.  16).  Defense  contractors 
are  making  important  decisions  regarding  system  requirements  and  design,  and  in  many 
cases,  defense  contractors  decide  who  will  be  developing  system  subcomponents  (GAO, 
2005).  This  environment  can  add  risk  to  DOD  acquisition  programs  if  not  properly 
managed  (Hillson,  2004,  p.  13). 


GAO’s  KAT 

The  GAO  has  recommended  that  the  DOD  implement  a  Knowledge-Based 
Acquisition  approach  since  1998.  In  1998  and  again  in  2002,  the  GAO  testified  before 
Congress  that  their  research  showed  that  commercial  industry  had  achieved  better  new- 
product  development  outcomes  by  developing  products  using  only  proven  technology, 
completed  product  designs,  and  by  having  their  production  processes  under  statistical 
control  (GAO,  1998;  GAO,  2002).  However,  over  the  past  ten  years,  program  managers 
have  not  fully  adopted  the  GAO’s  guidance  into  their  decision  making  process  based  on 
the  results  of  the  GAO’s  surveys  (GAO,  2009).  Why?  Has  the  “persistent  nature  of 
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acquisition  problems”  overwhelmed  decision  makers,  making  them  complacent  (GAO, 
2005)?  Is  the  DOD  so  unique  as  to  be  unable  to  achieve  results  similar  to  successful 
commercial  firms? 


DOD  versus  Commercial  Industry 

There  is  a  significant  difference  between  the  DOD’s  pursuit  of  new-product 
development  versus  that  of  the  commercial  sector.  One  such  difference  is  the  amount  of 
responsibility  each  program  manager  has  for  program  outcomes.  Program  managers  in 
the  companies  the  GAO  visited  were  incentivized  to  be  realistic  about  their  ability  to 
meet  their  program  goals  since  these  managers  were  likely  to  stay  with  a  program  from 
concept  to  production  (GAO,  2000,  p.  1).  As  a  result,  commercial  program  managers  are 
probably  less  likely  to  be  overly  optimistic  about  their  program  metrics. 

Department  of  Defense  program  managers  have  short  tenures  relative  to  their 
commercial  counterparts.  The  program  manager  who  establishes  the  initial  cost  and 
schedule  expectations  is  usually  not  accountable  for  achieving  those  expectations  (GAO, 
2000,  p.  9;  Deloitte  Consulting  LLP,  2008).  This  environment  holds  managers 
accountable  to  near  term  program  issues,  but  they  are  less  accountable  for  the  long-term 
viability  of  their  programs.  We  do  not  suggest  that  government  managers  are  dishonest, 
merely  that  they  are  incentivized  to  be  more  focused  on  the  near  term  aspects  of  the 
program  and  less  on  the  long-term  impacts  of  their  decisions. 

Furthermore,  it  is  likely  that  a  commercial  manager’s  promotion  potential  is  tied 
more  to  the  successes  and  failures  of  the  programs  they  have  managed  than  is  the 
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promotion  potential  of  a  DOD  manager.  As  a  result,  the  commercial  program  manager  is 
less  likely  to  take  on  undue  risk  or  attempt  to  inflate  the  reputation  of  their  programs 
(GAO,  2002,  pp.  3-10;  GAO,  2000,  p.  6).  On  the  other  hand,  DOD  program  managers 
promotion  opportunities  are  often  tied  less  to  the  long-term  performance  of  their  program 
and  more  to  its  performance  during  their  tenure  (GAO,  2000,  pp.  2-3).  Department  of 
Defense  program  managers  typically  change  many  times  throughout  the  life  of  the 
program  making  it  hard  to  associate  any  single  success  or  failure  to  a  specific  individual. 

Additionally,  a  DOD  program  manager  is  motivated  to  obtain  funding  each  year, 
and  to  be  an  advocate  for  their  program  and  the  user  (GAO,  2000,  p.  3;  Deloitte 
Consulting  LLP,  2008,  pp.  15-16).  Managers’  incentives  are  to  take  on  risk,  and  to 
pursue  an  aggressive  budget,  schedule,  and  performance  goal.  The  importance  of  this  is 
that  a  commercial  program  manager  may  view  the  Knowledge-Based  Approach  as  an 
appropriate  way  to  ensure  successful  outcomes,  while  a  DOD  program  manager  may  see 
the  Knowledge-Based  Approach  as  a  hindrance  that  slows  down  the  pace  of  his  or  her 
program. 

In  addition  to  having  different  incentives  for  their  managers,  DOD  and 
commercial  firms  pursue  new  product  development  for  different  reasons.  The  DOD’s 
mission,  to  supply  national  defense,  requires  that  it  constantly  develop  new  products  for 
use  in  maintaining  its  military  supremacy.  The  DOD  does  not  seek  to  profit  like 
commercial  business;  however,  it  is  motivated  to  minimize  cost  because  of  congressional 
pressure  and  a  limited  budget.  Commercial  firms  are  motivated  by  profit  generation, 
which  also  demands  that  they  limit  their  costs  in  order  to  be  successful.  Consequently, 
commercial  firms  initiate  new  product  development  only  when  a  solid  business  case 
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exists,  or  a  market  need  exists.  Commercial  firms  initiate  new  product  development 
when  they  are  certain  they  can  produce  a  product  in  a  relatively  short  time  period  and 
when  the  technology  needed  for  the  product  has  already  been  demonstrated  (GAO,  1998, 
pp.  5-7;  GAO,  2000,  p.  4).  Therefore,  while  the  incentive  structure  and  reasons  are 
different,  the  DOD  and  Commercial  Industry  both  seek  to  minimize  the  cost  and  time 
needed  to  produce  a  new  product. 


GAO’s  Knowledge-Based  Acquisition  Theory 

Our  analysis  focused  on  testing  the  assumptions  of  the  GAO’s  Knowledge-Based 
Acquisition  Theory  (KAT).  In  1998,  at  the  behest  of  Congress,  the  GAO  studied  whether 
or  not  “commercial  practices  offer  ways  to  improve  DOD’s  process  for  transitioning 
weapons  from  development  to  production  (GAO,  1998).”  The  subsequent  GAO  report 
focused  on  three  things.  First,  it  compared  DOD’s  practices  for  preparing  a  weapon 
system  for  production  and  the  best  practices  used  by  commercial  firms  for  similar 
projects.  Second,  it  examined  how  the  environments  that  the  DOD  and  commercial  firms 
operate  in  affect  their  new  product  development  practices.  Lastly,  the  report  discussed 
environmental  changes  that  the  GAO  considered  “key  to  the  success  of  DOD  initiatives 
for  improving  the  transition  of  weapons  from  development  to  production”  (GAO,  1998). 
The  purpose  of  GAO’s  study  was  to  help  the  DOD  find  effective  ways  of  expediting 
acquisition  schedules  and  to  help  programs  become  more  effective  (GAO,  1998).  The 
GAO  wanted  to  help  find  a  way  for  the  DOD  to  get  product  development  results  similar 
to  those  found  in  industry. 
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There  is  a  significant  difference  between  the  DOD  and  commercial  sectors 
management  of  the  technology  used  in  their  new-product  development  (GAO,  1998,  p. 
12).  Commercial  companies  make  a  distinction  between  technology  development  and 
product  development;  they  keep  the  two  activities  independent  of  one  another  (GAO, 
1998,  p.  12).  Commercial  new-product  development  includes  designing  and 
manufacturing  of  a  particular  product  that  meets  the  needs  of  a  particular  market. 


Figure  2:  Achieved  Technology  Development  (GAO,  1998) 


Conversely,  new-technology  development,  independent  of  new-product 
development,  looks  to  advance  current  technology  so  that  it  will  be  available  for  use  in 
future  products.  Commercial  firms  separate  technology  and  new  product  development 
because  they  view  technology  development  as  volatile  and  risky,  and  they  expect  it  will 
require  a  lot  of  rework  (GAO,  1998,  pp.  12-14;  GAO,  2000,  p.  4).  In  the  DOD, 
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technology  development  and  product  development  typically  occur  in  unison  as  part  of 
one  acquisition  strategy;  the  technology  that  is  developed  focuses  on  a  specific  need  for  a 
specific  product.  The  DOD  takes  on  the  risk  inherent  in  new  technology  development 
during  new  product  development. 

Figure  two  shows  how  differently  commercial  firms  and  the  DOD  approach 
technology  development  completion  (Knowledge  Point  1)  with  respect  to  new  product 
development  (program  launch).  The  commercial  firms  have  completed  technology 
development  prior  to  new-product  development,  whereas  the  DOD  will  continue  to 
develop  its  technology  well  into  production  (GAO,  1998,  pp.  14,22;  GAO,  2005,  p.  7). 

Commercial  practices  may  not  be  readily  adaptable  to  the  DOD’s  acquisition 
environment.  Unlike  commercial  organizations,  the  DOD  develops  cutting-edge 
products  that  test  the  limits  of  technology  so  that  it  can  maintain  military  supremacy. 
Further,  because  the  DOD  already  has  “first-class”  products  in  its  inventory,  new 
products  must  be  even  more  advanced  than  the  current  inventory  to  be  developed 
(Friedman,  2009).  Nevertheless,  if  the  DOD  seeks  program  outcomes  that  are  similar  to 
those  found  in  successful  commercial  firms,  it  may  benefit  from  studying  their  methods. 
The  GAO’s  Knowledge -Based  Approach  derived  from  their  research  of  successful 
commercial  firm  practices. 


Knowledge  Point  1:  Mature  Technology 

The  GAO’s  knowledge  point  one  is  synonymous  with  Milestone  B  for  DOD 
programs,  and  it  is  the  first  of  the  three  KAT  criteria  that  the  GAO  expects  will  lead  to 
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“better  weapon  system  outcomes”  (GAO,  1998).  A  product  must  have  obtained  a 
technology  readiness  level  of  seven  ( TRL7)  at  or  before  KP1  in  order  to  meet  the  GAO’s 
best  practice  standard.  In  other  words,  a  product’s  technology  must  work  in  its  intended 
operational  environment  (see  appendix  for  complete  TRL  definitions).  This  is  different 
from  current  DOD  practices,  as  current  policy  requires  programs  to  have  met  TRL6. 
Technology  Readiness  Level  6  demonstrates  technology  maturity  at  the  subcomponent 
level,  not  the  integrated  system  (USD(AT&L),  2008). 

According  to  the  GAO,  successful  commercial  firms  match  mature  technology 
with  system  requirements  at  KP1  and  then  proceed  to  produce  the  new  product. 
Successful  commercial  firms  do  not  usually  develop  a  new  technology  for  a  new  product 
since  technology  development  is  expensive  and  time  consuming.  Instead,  they  design 
new  products  around  pre-existing  and  mature  technology  (GAO,  1998). 

In  2006,  Congress  required  that  Major  Defense  Acquisition  Programs  (MDAPs) 
demonstrate  TRL6  prior  to  obtaining  milestone  B  approval  (Office  of  Air  Force  Lessons 
Learned  (HQ  USAF/A9L),  2009).  This  requirement  is  similar  to  the  recommendation 
made  by  the  GAO;  however,  the  difference  between  TRL  6  and  TRL  7  is  significant. 
According  to  the  GAO,  TRL  7  “represents  a  major  step  up  from  TRL  6 as  it  signifies 
that  the  prototype  works  in  an  operational  environment  on  an  actual  system  (GAO-09- 
326SP).  The  standard  for  TRL  6  is  that  a  prototype  works  in  a  realistic  environment  such 
as  a  “high  fidelity  lab”  or  a  “simulated  realistic  environment”  (GAO-09-326SP). 
Similarly,  The  Technology  Readiness  Assessment  (TRA)  Deskbook  states  that 
technology  should  be  proven  in  either  a  relevant  or  an  operational  environment  before  a 
program  can  proceed  into  system  development  (DUSD(S&T),  2005).  Furthermore,  the 
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Deputy  Under  Secretary  of  Defense  for  Acquisition  and  Technology  (DUSD  S&T) 
defines  technology  readiness  assessments,  as  a  “systematic,  metrics-based  process  and 
accompanying  report  that  assesses  the  maturity  of  certain  technologies  [  ]  used  in 
systems”  (DUSD(S&T),  2005,  pp.  1-2).  The  requirement  listed  in  the  TRA  Deskbook  is 
that  assessments  occur  for  each  increment  of  the  system  under  the  principles  of 
evolutionary  acquisition  strategy.  In  addition,  it  suggests  that  acquisition  programs 
should  deliver  timely  capability  to  the  warfighter  even  if  it  requires  producing  an  asset 
that  meets  only  part  of  the  user’s  need.  The  user  should  resist  the  urge  to  demand  design 
solutions  that  require  high-risk  requirements  (Office  of  the  Assistant  Secretary  of  the  Air 
Force,  2009).  We  designed  our  first  hypothesis  to  test  the  validity  of  the  assumptions 
surrounding  Knowledge  Point  1  (Figure  3). 

Hypothesis  1:  Defense  acquisition  programs  that  do  not  reach  Technology 
Readiness  Level  7  (TRL7)  by  program  initiation  (Milestone  B)  will  experience 
worse  program  outcomes  than  defense  acquisition  programs  that  do  reach  TRL7 
by  program  initiation. 

Knowledge  Point  2:  Product  Design 

The  second  criteria  a  program  must  meet  to  experience  better  outcomes  is 
Knowledge  Point  Two,  KP2.  This  Knowledge  Point  is  based  on  the  idea  that  mature 
product  design  will  reduce  program  uncertainty  and  in  turn,  increase  program  success. 

The  GAO  reported  in  1998  that  successful  commercial  firms  had  a  good  understanding  of 
their  product’s  design  (90%  of  Engineering  Drawings  Complete).  Additionally,  they 
posited  that  a  program  manager  conveys  confidence  in  a  product’s  ability  to  perform,  and 
the  maturity  level  of  the  product,  by  releasing  the  design  drawings  to  the  manufacturer 
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(GAO,  1998).  These  design  drawings  communicate  the  details  of  the  product  and  the 
materials  used,  as  well  as  how  the  products  piece/parts  performed  during  testing.  The 
heuristic  used  by  both  DOD  and  industry  is  that  the  product’s  design  is  essentially 
complete  when  roughly  90  percent  of  the  engineering  drawings  are  complete.  The 
percent  of  drawings  complete  is  determined  at  the  Critical  Design  Review  for  DOD 
programs.  In  2002,  the  GAO  reiterated,  “the  most  problematic  programs. .  .started 
production  before  design  and  manufacturing  development  work  was  concluded”  (GAO, 
2002,  p.  24).  However,  currently  the  majority  of  programs  do  not  achieve  KP2  on  time. 
We  developed  our  second  hypothesis  to  test  the  validity  of  the  assumptions  surrounding 
KP2  (Figure  3). 

Hypothesis  2:  Defense  acquisition  programs  that  do  not  complete  90%  of  their 
engineering  drawings  by  Critical  Design  Review  will  experience  worse  program 
outcomes  than  defense  acquisition  programs  that  do  complete  90%  of  their 
engineering  drawings  by  Critical  Design  Review. 


Knowledge  Point  3:  Production  Maturity 

Knowledge  Point  Three  ( KP3 )  is  the  third  component  that  the  GAO  found  critical 
to  the  success  of  commercial  firms.  To  meet  KP3  criteria  a  program’s  production 
processes  must  be  under  “statistical”  control  (GAO,  2002,  p.  13).  The  leading 
commercial  companies  the  GAO  visited  knew  that  they  could  produce  their  products 
within  their  quality  constraints  before  they  began  producing  production  articles  (GAO, 
1998).  These  firms  had  production  processes  within  statistical  control,  meaning  they 
were  able  to  produce  them  within  acceptable  deviation  tolerances.  According  to  the 
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GAO,  programs  should  meet  KP3  criteria  at  or  prior  to  the  start  of  production  (Milestone 
C).  Currently,  the  DOD  does  not  require  programs  to  have  production  processes  within 
statistical  control  at  the  Milestone  C  decision.  Production  Qualification  does  not  occur 
until  after  the  Milestone  C  decision. 

Maintaining  consistency  during  the  production  phase  is  critical  because  deviations 
and  defects  found  after  manufacturing  a  product  will  cause  rework  and  can  increase  the 
cost  of  each  unit.  The  GAO’s  KP3  measurement  provides  an  assessment  of  the 
manufacturer’s  production  ability,  and  meeting  it  should  help  prevent  rework  associated 
with  production  process  issues.  Additionally,  since  DOD  products  are  manufactured 
externally,  program  managers  can  use  KP3  to  monitor  the  progress  of  the  contractor  and 
subcontractor.  We  designed  our  final  hypothesis  to  test  the  validity  of  the  assumptions 
surrounding  KP3  (Figure  3). 


Hypothesis  3:  Defense  acquisition  programs  that  do  not  have  manufacturing 
processes  stabilized  by  Milestone  C  will  experience  worse  program  outcomes 
than  defense  acquisition  programs  that  do  have  their  manufacturing  processes 
stabilized  by  Milestone  C. 


KP1  KP2  KP3 


Technology 

System  Development 

Production  & 

Development 
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MSA  MSB  CDR  MSC 


Figure  3:  Knowledge  Point  Schedule 


31 


Current  Acquisition  Reform 

In  late  2005,  the  GAO  conveyed  its  continuing  concerns  about  the  DOD’s 
management  of  its  investment  portfolio  to  the  U.S.  Senate.  The  GAO’s  concern  was  that 
acquisition  program  cost  increases,  combined  with  operations  in  Iraq  and  Afghanistan 
will  prohibit  the  military  from  delivering  assets  essential  to  the  warfighter  (GAO,  2005). 
Further,  they  said  that  the  DOD  knows  what  to  do  to  gamer  more  successful  acquisition 
outcomes  but  that  it  is  not  implementing  the  controls  necessary  to  initiate  the  change 
(GAO,  2005).  In  addition,  the  GAO  was  concerned  that  the  typical  margin  of  error  in 
percentage  terms  will  equate  to  significant  cost  overruns  as  the  acquisition  portfolio 
grows  in  value.  According  to  GAO,  in  a  mere  five  years  (2001  to  2006)  the  Total  Cost  of 
the  DOD’s  top  five  acquisition  programs  grew  from  $291  Billion  Dollars  to 
approximately  $550  Billion  Dollars  as  shown  in  their  table,  reproduced  below  (Table  1). 


Table  1:  Total  Cost  of  DOD's  Top  5  Programs  (FY06  Dollars) 


2001 

2006 

Program 

Cost 

Program 

Cost 

F/A-22  Raptor  Aircraft 

$65.0  billion 

Joint  Strike  Fighter 

$206.3  billion 

DDG-5 1  class  destroyer  ship 

$64.4  billion 

Future  Combat  Systems 

$127.5  billion 

Virginia  class  submarine 

$62.1  billion 

Virginia  class  submarine 

$80.4  billion 

C-17  Globemaster  airlift 
aircraft 

$51.1  billion 

DDG-5 1  class  destroyer  ship 

$70.4  billion 

F/A-18E/F  Super  Hornet 
fighter  Aircraft 

$48.2  billion 

F/A-22  Raptor  aircraft 

$65.4  billion 

Total 

$290.8  billion 

Total 

$550.0  billion 

(Adapted  from  (GAO,  2005,  p.  3)) 
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Weapon  System  Reform  Act  of 2009 

GAO’s  KAT  has  influenced  the  acquisition  community  for  the  past  ten  years,  most 
recently  affecting  the  policy  enacted  in  the  2009  Weapon  Systems  Acquisition  Reform 
Act  (11 1-23,  2009).  Recent  changes  in  DOD  acquisitions  essentially  direct  the  DOD 
acquisition  community  to  implement  the  KAT  approach.  On  May  22,  2009  the  1 1 1th 
Congress  passed  the  Weapon  Systems  Acquisition  Reform  Act  of  2009  -  Public  Law  111- 
23,  to  improve  the  DOD  acquisition  environment.  This  Act  specifically  prescribes 
changes  to  acquisitions  through  three  general  categories:  (1)  The  Acquisition 
Organization,  (2)  Acquisition  Policy,  and  (3)  Additional  Acquisition  Provisions.  This 
Reform  Act  seems  to  be  consistent  with  the  GAO’s  KAT,  it  generally  supported  the 
GAO’s  claim  that  following  the  KAT  would  produce  better  program  outcomes. 

First,  Section  103  of  the  Act,  Performance  Assessments  and  Root  Cause  Analysis 
for  Major  Defense  Acquisition  Programs,  states  that  the  Secretary  of  Defense  must 
designate  a  senior  official  who  will  conduct  performance  assessments  and  root  cause 
analysis  for  MDAPs  ((11 1-23,  2009)).  In  other  words,  they  will  investigate  MDAPs  that 
miss  one  or  more  key  performance  parameters  to  determine  why  they  missed  the 
parameter.  The  person  in  this  position  will  be  responsible  for  the  performance 
assessments  of  MDAPs  on  a  periodic  and  an  as  needed  basis.  They  will  conduct  root 
cause  analyses  of  MDAPs  as  well  as  issue  the  policy  and  guidance  needed  to  govern  the 
process  of  performing  these  root  cause  analyses.  This  analysis  should  yield  an  evaluation 
of  the  usefulness  of  the  current  metrics  used  to  assess  program  performance  and, 
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recommend  any  necessary  changes  to  the  metrics.  Lastly,  this  position  will  advise 

acquisition  officials  on  potential  program  performance  issues. 

Again,  the  purpose  of  the  Root  Cause  Analysis  is  to  assess  the  underlying  cause 

of  a  programs  inability  to  meet  one  or  more  key  performance  parameters  (PL1 11-13). 

The  parameters  include  cost,  schedule,  and  performance  as  well  as  the  following  eight 

additional  parameters: 

I.  Unrealistic  performance  expectations 

II.  Unrealistic  baseline  estimates  for  cost  or  schedule 

III.  Immature  Technology  or  excessive  manufacturing  or  integration  risk 

IV.  Changes  in  Procurement  Quantities 

V.  Inadequate  program  funding  or  funding  Instability 

VI.  Poor  Performance  by  Government  or  Contractor  Personnel  responsible  for 
program  management 

VII.  Any  other  Matters 

Of  particular  significance  in  the  context  of  this  study,  is  item  III  from  the  list, 
Immature  Technology  or  excessive  manufacturing  or  integration  risk.  The  2009  GAO 
report  found  that  of  the  39  programs  that  provided  them  with  data  only  14  had,  or  would 
have,  mature  technology  at  or  prior  to  Milestone  B  (GAO  2009,  pg  16).  We  expect  that  if 
having  mature  technology  at  MSB  does  provide  better  program  outcomes  then  the  root 
cause  analysis  will  substantiate  the  GAO’s  claim. 

Section  104  of  Public  Law  111-23  requires  the  Directors  of  Defense  Research  and 
Engineering  and  Developmental  Test  and  Evaluation  to  submit  a  report  of  technology 
maturity  and  integration  risk  to  the  Secretary  of  Defense  annually.  This  change  is 
consistent  with  the  GAO’s  KAT,  specifically  KP 1  (technology  maturity  level).  This 
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section  requires  that  the  Director  of  Defense  Research  and  Engineering  develop  and 
publish  knowledge -based  technology  maturity  standards  for  MDAPs.  It  requires  that 
technology  maturity  assessments  occur  at  key  stages  in  the  acquisition  process.  We 
expect  that  since  H.R.  2101  amended  Title  10  moving  the  Preliminary  Design  Review 
process  prior  to  the  Milestone  B  decision,  that  the  technology  maturity  measurement  will 
also  happen  prior  to  Milestone  B. 

In  the  1980’s  acquisition  reform  focused  on  fraud,  waste,  and  abuse.  The  Packard 
Commission  altered  that  trajectory  to  a  focus  on  increasing  the  efficiency  and 
effectiveness  of  DOD  acquisitions.  Nevertheless,  DOD  acquisition  programs  still  suffer 
from  cost  and  schedule  overruns.  We  hope  that  through  our  assessment  of  the  KAT  we 
can  determine  if  programs  that  met  the  KP  criteria  as  outlined  by  the  GAO  experienced 
better  program  outcomes  than  programs  that  did  not.  We  anticipate  that  the  results  of  our 
analysis  will  help  DOD  program  managers  better  understand  the  effectiveness  of  the 
KAT. 


35 


Chapter  III:  Data  Collection  and  Methodology 


Data  Source  and  Variables 

We  used  acquisition  program  performance  information  for  our  analysis.  We 
collected  this  information  from  assessments  that  the  Government  Accountability  Office 
conducted  and  published.  These  reports  documented  acquisition-program  performance 
characteristics  for  high  profile  and  large  budget  acquisition  programs  (GAO,  2008,  p.  3). 
The  GAO  performed  these  studies  and  published  their  reports  annually  from  Calendar 
Year  2003  through  Calendar  Year  2009,  with  each  report  covering  an  average  of  55 
programs.  We  conducted  approximately  twenty  random  data  point  verifications  to 
validate  the  accuracy  of  our  data. 

We  used  the  most  recent  iteration  of  each  program  found  in  the  GAO  reports  for 
our  analysis.  We  used  only  the  most  current  instance  of  each  program  to  minimize  the 
intercorrelation  of  our  predictor  variables  (Kutner,  Nachtsheim,  Neter,  &  Li,  2004,  p. 
279).  This  yielded  a  sample  size  of  107  acquisition  programs.  Unfortunately,  not  all 
programs  had  all  of  the  appropriate  data  reducing  the  sample  size  further  for  some  of  our 
tests.  For  example,  there  were  90  programs  with  complete  data  for  KP1,  71  programs 
with  complete  data  for  KP2,  and  51  programs  with  complete  data  for  KPS.  Furthermore, 
of  the  programs  with  complete  data,  only  11  programs  had  met  the  KP1  criteria,  only  15 
had  met  KP2  criteria,  and  only  4  programs  had  met  KPS  criteria. 

We  used  the  program  performance  data  that  were  contained  in  the  GAO  reports  to 
test  our  hypotheses.  To  recap,  the  GAO’s  claim  is  that  if  a  program  meets  the 
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knowledge-point  ( KP )  criteria  at  the  predetermined  junctures,  the  program  will 
experience  better  results  than  programs  that  did  not  meet  the  KP  criteria.  We  considered 
the  knowledge  points  “treatments,”  and  compared  the  outcomes  of  the  programs  that  met 
the  KP  criteria  with  those  that  did  not.  We  expected  to  find  cost  and  schedule  increases 
to  be  smaller  for  programs  that  met  the  KP  criteria,  and  we  expected  to  find  a  smaller 
reduction  in  the  number  of  units  produced  for  programs  that  met  the  KP  criteria. 


Ho.  p  outcome  of  Group  0  >  p  outcome  of  Group  1 
Ha:  p  outcome  of  Group  0  <  p  outcome  of  Group  1 

Where  Group  0  did  not  meet  KP  criteria  and  Group  1  Met  KP  criteria 
DVs/Outcomes  measured:  QCP 

Ho.  p  outcome  of  Group  0  <  p  outcome  of  Group  1 
Ha:  p  outcome  of  Group  0  >  p  outcome  of  Group  1 

Where  Group  0  did  not  meet  KP  criteria  and  Group  1  Met  KP  criteria 
DVs/Outcomes  measured:  UCC,  TCC,  RDCC,  SIP,  PCC 


Figure  4:  Knowledge  Point  Attainment  (GAO  2008) 
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The  GAO  measured  whether  or  not  a  program  met  KP1,  KP2,  and  KPS  criteria  at 
the  KAT’s  predetermined  knowledge  points  (Figure  4)  in  each  report.  We  used  these 
KP1 .  KP2 ,  and  KPS  measurements  as  our  independent  variables.  We  gave  each 
independent  variable  an  indicator  of  “0”  if  the  program  did  not  meet  the  KAT  criteria  on 
time  and  gave  a  “1”  to  programs  that  did  meet  the  KAT  criteria  on  time. 


Limitations  of  the  Data 

There  are  many  factors  external  to  the  program  office  that  can  affect  program 
outcomes.  For  instance,  unexpected  changes  to  program  requirements,  volatile  funding 
sources,  and  unexpected  schedule  changes  due  to  factors  beyond  the  control  of  the 
program  office  can  disrupt  program  performance  outcomes  (GAO,  Weapons  Acquisition: 
A  Rare  Opportunity  for  Lasting  Change,  1992).  These  external  factors  may  limit  the 
explanatory  power  of  our  models  since  they  focused  solely  on  whether  a  program  met  the 
KP  criteria  (Deloitte  Consulting  LLP,  2008;  Kutner,  Nachtsheim,  Neter,  &  Li,  2004,  p. 
74). 

Our  sample  size  of  107  programs  was  generally  enough  to  assess  whether  a 
statistically  significant  relationship  existed  between  programs  that  met  the  KP  criteria 
versus  those  that  did  not  (Kutner,  Nachtsheim,  Neter,  &  Li,  2004).  However,  many 
programs  have  not  met  the  KP  criteria  on  time.  This  limited  the  generalizeability  of  our 
findings  for  other  programs  since  such  a  small  sample  could  contain  atypical  program 
outcomes  relative  to  the  general  program  population  (Trochim  &  Donnelly,  2008,  p.  36). 
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In  addition,  we  have  many  more  data  points  for  KP1  than  for  KP2  or  KP3  due  to  the  time 
series  nature  of  our  data.  Many  programs  were  not  far  enough  along  to  have  a 
measurement  for  KP2  or  KP3.  The  disparity  in  our  data  was  reason  for  us  to  conduct 
independent  KP  analysis  so  we  could  leverage  the  data  points.  If  we  had  analyzed  the 
KP'%  in  combination,  we  would  have  eliminated  many  data  points  due  to  missing  data. 

In  a  few  circumstances,  we  made  assumptions  based  on  the  data  available.  Some 
program  assessments  lacked  information  about  whether  a  program  met  a  given  KP  on 
time.  However,  these  reports  did  assess  the  programs  current  KP  status.  If  a  program  did 
not  meet  a  KP  as  of  the  report  date,  and  the  report  date  was  post  KP  measurement,  we 
assumed  the  program  had  not  met  the  preceding  KP  criteria  on  time.  Figure  5  exhibits 
this  scenario.  This  program  did  not  have  the  data  available  regarding  KP1,  but  it  is 
reasonable  to  assume  that  since  they  had  not  met  the  criteria  in  2007  that  they  had  not 
met  the  criteria  in  1995. 


Figure  5:  Data  Deduction  (GAO,  2008,  p.  27) 
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The  GAO  did  not  randomly  select  the  acquisition  programs  they  followed 
(Trochim  &  Donnelly,  2008,  p.  38).  However,  our  dataset  contains  a  good  mixture  of  the 
different  program  types  (Table  2).  We  consider  these  programs  to  be  a  good 
representation  of  DOD  acquisition  programs  and  represent  the  environment  in  which  the 
GAO  expects  the  KAT  to  apply. 


Table  2:  Data  Demographics 


Number  of  Programs 

Percent  of  Dataset 

Air 

33 

31% 

Space 

15 

14% 

Munition 

11 

10% 

Sea 

15 

14% 

Missile 

16 

15% 

MAIS 

14 

13% 

Land 

2 

2% 

Space/Air 

1 

1% 

Dependent  Variables 

We  collected  the  dependent  variable  (DV)  information  from  the  same  GAO 
reports  that  contained  our  independent  variables.  The  information  we  gathered  measured 
specific  program  performance  parameters  such  as  schedule,  cost  and  quantity  change. 
Since  the  GAO’s  KAT  associates  KP  completion  with  better  acquisition  program 
outcomes,  these  variables  provided  a  way  for  us  to  measure  program  performance.  The 
performance  measures  we  used  were  Quantity  Change  Percent  ( QCP ),  Unit  Cost  Change 
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( UCC ),  Total  Cost  Change  ( TCC ),  R&D  Cost  Change  ( RDCC ),  Schedule  Increase 


Percentage  (SIP),  and  Procurement  Cost  Change  ( PCC)  (Figure  6). 


Concept 

System  development 

Production 

A 

A 

▲ 

A 

▲ 

▲ 

▲ 
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Dwelopment 

Design 

Production 

GAO 

First 

It  M 

start 
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review 

launch 

capability 

(4/99) 

(9/01) 

(4/04) 

(6»4) 

11/00) 

(11/06) 

(6/10) 

Program  Essentials 

Program  Performance  (fiscal  year  2008  dollars  in  millions) 

Prime  contractor:  Lockheed  Martin 

as  or 

Latest 

Percent 

Program  office:  El  Segundo.  Calif. 

10/2001 

12/2006 

change 

Funding  needed  to  complete 

Research  and  development  cost 

S4 ,669.0 

$6,098.9 

30.6 

R&D:  *1.078.9  million 

Procurement  cost 

*1380.9 

S718.9 

-47.9 

Procurement:  $93.6  million 

Total  program  coat 

S6.050.0 

S6.817.3 

12.9 

Total  funding  *1.172.9  million 

Program  unit  cost 

*1 ,209.993 

*2.272.443 

87.8 

Procurement  quantity:  0 

Total  quantities 

S 

3 

-40.0 

Acquisition  cycle  time  (months) 

111 

134 

20.7 

Figure  6:  Program  Performance  with  Respect  to  Program  Baseline  (GAO,  2008) 


As  previously  mentioned,  the  GAO  claimed  that  programs  that  met  the  KAT 
criteria  would  experience  better  acquisition  outcomes  than  programs  that  did  not.  Since 
program  sizes  vary,  we  controlled  for  program  size  by  using  percentages.  We  expect 
better  performance  to  be  evident  through  smaller  percentage  increases  from  a  programs 
baseline.  We  defined  a  better  acquisition  outcome  to  mean  an  outcome  in  which  a 
program  performs  at  or  below  its  program  baseline,  or  to  mean  better  performance  with 
respect  to  those  programs  that  did  not  meet  the  KP  criteria. 

Dependent  Variable  Description 

The  first  Dependent  Variable  (Table  3)  we  used  was  Schedule  Increase  Percent 
(SIP),  this  variable  measured  program  schedule  change  with  respect  to  a  program’s 
baseline.  To  test  the  KAT  claim,  we  tested  whether  programs  that  met  the  KP  criteria 
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performed  better  than  those  programs  that  did  not  meet  the  KP  criteria.  Better 
performance  would  be  indicated  by  a  smaller  percent  schedule  change. 

The  next  Dependent  Variable  we  used  was  Quantity  Change  Percent  ( QCP ). 

This  variable  measured  the  percentage  change  in  a  program’s  anticipated  production 
quantity  relative  to  the  number  the  program  expected  to  produce  at  its  baseline  date.  We 
reasoned  that  managers  at  risk  of  exceeding  their  budgets  might  need  to  reduce  their 
production  quantities  in  order  to  stay  within  budget.  Therefore,  we  expect  that  for  this 
variable,  smaller  quantity  reductions  indicate  better  program  outcomes.  The  exception  to 
this  would  be  if  quantities  increased  due  to  mission  need,  regardless  of  poor  unit  cost 
performance.  We  had  19  programs  in  our  dataset  that  had  not  met  any  KP  criteria  they 
were  accountable  for,  but  had  an  increase  in  QCP  nevertheless.  Of  these  programs,  there 
were  ten  programs  that  missed  the  KP  criteria  and  experienced  significant  unit  cost 
growth,  and  yet  these  programs  increased  the  number  of  units  they  produced.  This 
behavior  leads  us  to  question  the  degree  to  which  QCP  is  a  good  metric  for  “good” 
acquisition  programs.  Consequently,  we  rely  less  on  this  variable  than  some  of  our  other 
variables  which  we  think  are  more  highly  correlated  with  successful  acquisition 
programs. 

In  addition  to  quantity  change,  we  looked  at  the  change  in  unit  cost,  UCC.  We 
tested  to  see  if  programs  that  met  the  KP  criteria  had  a  smaller  unit  cost  percent  change 
than  programs  that  did  not  meet  the  KP  criteria.  We  collected  all  unit  cost  change 
amounts  in  percent  change  relative  to  their  estimated  cost  on  the  program  baseline  date. 

We  also  looked  at  Total  Program  Cost  Change,  TCC,  to  determine  if  a  program’s 
total  cost  change  was  smaller  for  programs  that  met  the  KP  criteria  than  for  programs  that 
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did  not.  Since  the  three  KP ’s  focus  on  different  types  of  knowledge  and  are  at  different 
points  in  the  program  lifecycle,  it  is  reasonable  to  assume  that  they  may  affect  particular 
components  of  a  program  more  than  others.  TCC  should  allow  us  to  see  the  total 
financial  effect  the  KPs  have  on  a  program  as  opposed  to  growth  in  one  particular  area. 

Procurement  Unit  Cost  is  the  cost  to  purchase  a  unit  (its  piece/parts  and  assembly 
cost)  whereas  a  program’s  unit  cost  is  unit  cost  as  a  function  of  total  program  costs  (Total 
cost/number  of  units).  We  used  the  Procurement  Unit  Cost  Change,  PCC,  to  determine  if 
programs  that  met  the  KP  criteria  had  a  smaller  increase  in  procurement  costs  relative  to 
programs  that  did  not  meet  the  KP  criteria. 

Our  last  variable,  Research  and  Development  Cost  Change,  RDCC,  measured  a 
program’s  R&D  cost  change  in  percentage  terms.  We  expected  that  programs  that  did 
not  have  mature  technology  and  an  understanding  of  product  design  would  experience 
research  and  development  cost  growth. 

The  GAO’s  KAT  states  that  programs  that  meet  the  KP  criteria  on  time  will 
experience  better  outcomes.  We  used  the  variables  UCC,  PCC,  TCC,  RDCC,  SIP,  and 
QCP  as  our  measures  of  program  performance. 

The  GAO  described  the  KAT  as  a  sequential  process  and  that  in  order  to  meet 
either  KP2  or  KP3  a  program  would  need  to  have  met  the  preceding  KP(s)  (GAO,  2004, 
pp.  4-5).  It  is  reasonable  to  assume  that  a  program  will  be  more  likely  to  achieve  either 
KP2  or  KPS  if  it  has  achieved  the  KPs  that  precede  it.  However,  we  found  that  eleven 
programs  in  the  reports  had  met  KP2  criteria  without  meeting  KP1  criteria.  In  addition, 
since  each  KP  was  measured  using  criteria  unique  to  that  KP  and  because  each  KP 
measurement  is  independent  of  the  other  KPs,  we  examined  each  KP  individually. 
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Methodology 

The  primary  purpose  of  our  study  was  to  test  the  GAO’s  KAT  by  determining  if  a 
difference  existed  between  the  outcomes  of  two  groups,  those  that  have  met  KP  criteria 
and  those  that  have  not.  We  compared  programs  that  met  KP1,  KP2,  and  KPS  with 
programs  that  did  not  to  see  if  programs  that  met  the  criteria  performed  better.  We 
conducted  Independent-Sample  T-Tests  to  determine  if  the  mean  performance  of  the 
programs  that  met  the  KP  criteria  was  better  than  the  mean  performance  of  the  programs 
that  did  not  (Kutner,  Nachtsheim,  Neter,  &  Li,  2004,  pp.  1309-1310).  We  used  this 
methodology  instead  of  an  ANOVA  because  our  research  question  involves  one-way  or 
directional  hypotheses,  which  indicate  that  F  tests  are  inappropriate.  T-Tests  are  an 
appropriate  method  for  determining  if  the  mean  of  the  group  that  failed  to  meet  KP1 
(coded  as  KP1=0)  is  “worse”  than  the  mean  of  the  group  that  achieved  KP1,  where 
“worse”  may  be  greater  or  lesser  depending  on  the  particular  metric  (Kutner,  Nachtsheim, 
Neter,  &  Li,  2004,  p.  1310). 

Our  primary  statistical  test  was  the  standard  T  test  with  pooled  variance  (Bulmer, 
1979,  pp.  145-154;  McClave,  Benson,  &  Sincich,  2008).  One  of  the  assumptions  of  this 
test  is  that  both  groups  have  the  same  variance.  To  test  this  assumption  we  conducted  the 
Levene  Test  and  examined  the  corresponding  P-Value  (McClave,  Benson,  &  Sincich, 
2008,  p.  455).  In  cases  where  this  P-value  was  less  than  0.05,  we  concluded  that  the 
variances  were  statistically  different  and  consequently  the  standard  T  test  was 
inappropriate.  In  these  cases,  we  instead  used  the  Welch  Test  since  it  accounts  for 
different  variances  between  the  groups  (Montgomery,  1999,  p.  392).  In  either  case 
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(standard  T  test  or  Welch  test),  we  examined  the  resulting  P-value  to  find  significant 


relationships. 


Table  3:  Variable  List 


Description 

Nomenclature 

Dependent 

Variables 

Schedule  Increase 

Percent 

Measured  the  schedule  increase  relative  to  the 
Program’s  Baseline  Date  (Percentage) 

SIP 

Quantity  Change 
Percent 

Measured  an  increase/decrease  in  production 
units  relative  to  the  Program’s  Baseline  Date 
(Percentage) 

QCP 

Unit  Cost  Change 
Percent 

Measured  the  change  in  cost  of  each 
production  unit  relative  to  the  Program’s 
Baseline  Date  (Percentage) 

ucc 

Total  Cost  Change 
Percent 

Measured  the  change  in  the  total  cost  of  a 
program  relative  to  the  Program’s  Baseline 

Date  (Percentage) 

TCC 

Procurement  Cost 
Change  Percent 

Measured  the  change  in  Procurement  Costs 
relative  to  the  Program’s  Baseline  Date 
(Percentage) 

PCC 

Research  & 
Development  Cost 
Change 

Measured  the  change  in  Research  and 
Development  costs  relative  to  the  Program’s 
Baseline  Date  (Percentage) 

RDCC 

Independent 

Variables 

Knowledge  Point  1 

Measured  whether  a  program  had  mature 
technology  (Technology  Readiness  Level  7) 
by  program  initiation  (Milestone  B) 

KP1 

Knowledge  Point  2 

Measured  whether  a  program  had  90%  of  its 
engineering  drawings  by  Critical  Design 
Review 

KP2 

Knowledge  Point  3 

Measured  whether  a  program’s  manufacturing 
processes  were  in  statistical  control  by 
Milestone  C 

KP3 
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Distribution  of  Variables 


In  addition  to  testing  the  relationships  between  meeting  KP  criteria  and  program 
performance,  we  looked  at  the  distribution  of  our  variables.  We  visually  examined  the 
distribution  to  see  if  programs  that  met  the  KP  criteria  experienced  a  more  tightly 
distributed  range  of  outcomes  than  programs  that  did  not  meet.  In  other  words,  though 
programs  may  not  have  different  mean  outcomes  with  respect  to  meeting  KP  criteria, 
programs  that  meet  the  KP  criteria  may  have  a  smaller  deviation  from  the  mean  than 
programs  that  did  not  meet  the  KP  criteria  (Figures  8  -  10). 

Finally,  we  looked  at  the  effects  of  the  KAT  on  programs  by  type  of  program.  For 
example,  we  wanted  to  see  if  a  program’s  product  had  any  correlation  to  whether  it  met 
the  KP  criteria  or  if  the  KAT  was  more  effective  on  certain  types  of  products.  Our  sample 
size  was  too  small  for  statistical  analysis  so  we  conducted  a  visual  analysis  of  the  data  for 
this  measurement. 


Controlling  for  Program  Age 

We  supplemented  our  t-tests  with  an  additional  model  to  see  if  a  program’s  age 
influenced  our  results.  One  of  the  features  of  our  data  is  that  GAO’s  program 
perfonnance  measurements  occurred  at  different  points  in  the  program’s  lifecycles.  For 
example,  the  GAO  measured  some  programs  shortly  after  the  program  reached  a 
Knowledge  Point,  while  in  other  cases,  the  GAO’s  measurement  occurred  many  years 
after  a  program  had  reached  a  Knowledge  Point.  This  meant  that  some  programs  had 
much  more  time  to  incur  cost,  schedule,  and  quantity  change  than  other  programs  and 
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consequently,  we  concluded  that  by  adding  AGE  into  our  model  we  could  control  for 
these  differences  and  increase  the  fidelity  of  the  relationships. 

We  used  the  program  initiation  date  (Milestone  B)  and  the  date  of  the  GAO’s 
measurement  to  determine  the  approximate  number  of  days  a  program  had  existed 
(AGE).  We  then  retested  our  hypothesis  using  linear  regression  models  in  lieu  of  t-tests 
to  accommodate  our  control  variable.  The  model  chosen  was  a  straightforward  bivariate 
regression  model  of  the  form: 

Y  =  p0  +  ^KP  +  (32AGE  +  £t 

Y  Variables:  UCC,  QCP,  PCC,  SIP,  TCC,  RDCC;  Error  Term:  et 

In  the  course  of  fitting  these  regression  models,  we  conducted  reasonable 
regression  diagnostics  in  order  to  check  the  assumptions  of  ordinary  least-squares  (OLS) 
regression,  which  is  zero-mean,  independent,  constant  variance  error  terms,  as  well  as 
checks  for  multicolinearity  and  influential  points  (Kutner,  Nachtsheim,  Neter,  &  Li, 

2004,  pp.  102-103).  The  results  of  these  diagnostics  implied  non-constant  variance,  so  in 
response  we  utilized  the  standard  technique  of  transforming  the  Y 
variable  (Kutner,  Nachtsheim,  Neter,  &  Li,  2004, pp.  132  —  137).  In  most  cases,  a 
simple  natural  logarithm  transform  was  sufficient  to  remedy  the  non-constant  variance, 
although  for  RDDC  and  TCC,  Box-Cox  analysis  showed  that  a  power  transform  with 
A  —  —1.4  appeared  more  appropriate.  In  all  cases,  we  were  able  to  develop  a  model  that 
appeared  to  satisfy  the  assumptions  of  OLS. 
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Chapter  IV:  Analysis  and  Results 


Table  4  provides  the  descriptive  statistics  of  our  variables.  We  found  that 
missing  the  KP  criteria  did  not  have  a  pervasive  impact  on  our  dependent  variables. 

Some  dependent  variables  showed  improvement  when  programs  met  the  KPs,  but  most 
dependent  variables  did  not.  However,  we  were  still  able  to  find  some  important 
relationships  between  programs  meeting  the  KP’s  and  better  program  performance. 
Additionally,  we  assessed  whether  program  type  had  an  effect  on  a  program’s 
perfonnance  or  a  program’s  likeliness  to  meet  the  KP  criteria.  We  could  not  test  our 
conclusions  statistically,  but  still  found  the  results  relevant  to  this  study. 

Test  of  Hypotheses 

The  results  of  our  analyses  are  in  Table  5.  Through  our  analysis  of  these 
programs,  we  found  a  few  significant  relationships.  First,  we  found  that  Research  and 
Development  Cost  Change  related  closely  to  KP1.  Programs  that  had  mature  technology 
at  or  prior  to  Milestone  B  had  a  smaller  increase  in  Research  and  Development  costs  than 
programs  that  had  immature  technology  at  Milestone  B.  This  confirms  our  expectation 
that  RDCC  is  a  good  measure  of  program  performance,  specifically,  with  respect  to  KP  I . 
Furthermore,  we  presume  that  programs  with  immature  technology  have  a  higher  rate  of 
Research  and  Development  funding  cost  growth  since  technology  maturation  must  occur 
as  the  program  moves  past  Milestone  B. 
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PCC  8.  RDCC  7.  TCC  6.  UCC  5.  QCP  4.  SIP  3.  KP3  2.  KP2  1.  KP1 


Table  4:  Descriptive  Statistics  (one-tailed) 


Correlation  Matrix 


Mean 

STDV 

1 

2 

3 

4 

5 

6 

7 

8 

9 

Pearson  Correlation 

1.00 

.961" 

.946” 

-0.07 

.293" 

-0.20 

-0.09 

-0.17 

-0.05 

Sig.  (1 -tailed) 

1.19 

0.00 

0.00 

0.30 

0.01 

0.05 

0.24 

0.07 

0.34 

N 

91 

69 

42 

54 

67 

65 

72 

73 

65 

Pearson  Correlation 

1.00 

.948" 

-0.01 

.237 

-0.21 

-0.13 

-0.17 

-0.09 

Sig.  (1 -tailed) 

1.79 

0.00 

0.47 

0.04 

0.07 

0.17 

0.10 

0.26 

N 

72 

42 

44 

55 

54 

59 

59 

53 

Pearson  Correlation 

1.00 

-0.15 

.385" 

-.258' 

-0.06 

-0.04 

-0.09 

Sig.  (1 -tailed) 

0.61 

0.20 

0.01 

0.05 

0.36 

0.39 

0.29 

N 

52 

34 

43 

42 

44 

44 

40 

Pearson  Correlation 

1.00 

0.10 

.523" 

.331" 

.337" 

.242' 

Sig.  (1 -tailed) 

0.29 

0.50 

0.22 

0.00 

0.00 

0.00 

0.02 

N 

67 

66 

66 

67 

67 

66 

Pearson  Correlation 

1.00 

-.268“ 

.542" 

.481" 

.867" 

:  Sig.  (1 -tailed) 

0.26 

1.51 

0.01 

0.00 

0.00 

0.00 

N 

81 

79 

81 

81 

78 

Pearson  Correlation 

1.00 

0.14 

0.10 

0.00 

Sig.  (1 -tailed) 

0.48 

0.95 

0.10 

0.18 

0.49 

N 

79 

79 

79 

76 

Pearson  Correlation 

Sig.  (1 -tailed) 

N 

0.52 

1.17 

1.00 

86 

.822" 

0.00 

86 

.954" 

0.00 

79 

Pearson  Correlation 

1.00 

.716" 

Sig.  (1 -tailed) 

0.98 

4.16 

0.00 

N 

87 

79 

Pearson  Correlation 

1.00 

Sig.  (1 -tailed) 

N 

0.53 

1.27 

79 

**.  Correlation  is  significant  at  the  0.01  level  (1  -tailed). 
*.  Correlation  is  significant  at  the  0.05  level  (1 -tailed). 
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In  addition  to  RDCC,  we  found  that  the  Unit  Cost  Change  ( UCC )  for  programs 
that  did  not  meet  the  KP  criteria  increased  more  than  for  those  programs  that  met  the  KP 
criteria.  In  retrospect,  we  find  that  UCC  is  an  important  variable  since  the  cost  of  each 
unit  is  a  function  of  several  other  variables  (Figure  7).  We  find  UCC  to  be  more 
important  than  any  other  variable  because  it  is  affected  when  any  cost  or  quantity  variable 
changes.  Unit  Cost  Change  is  a  sensitive  variable  encompassing  many  program  changes. 
Our  finding  was  confirmed  given  that  this  variable  had  a  significant  relationship  with 
more  independent  variables  than  any  other  dependent  variable.  We  found  that  KP1,  and 
KP3,  related  closely  to  Unit  Cost  Change  and  that  KP2  had  a  tenuous  relationship. 

Lastly,  we  could  not  prove  that  if  a  program  met  the  KP  criteria  it  would 
experience  a  smaller  schedule  increase  than  if  it  did  not  meet  the  KP  criteria.  This 
finding  was  interesting,  because  it  seems  counterintuitive.  We  expected  to  find  that 
programs  that  missed  the  KP  criteria  would  experience  larger  schedule  increases  than 
programs  that  met  the  KP  criteria.  Specifically,  we  expected  programs  that  missed  KP1 
or  KP2  to  have  larger  schedule  increases  since  still  needed  to  mature  their  technologies 
and  complete  their  product  designs. 


T otal  Program  Cost  * 

Unit  Cost  = - - - 

Total  Quantities 


*  Total  Program  Cost  =  Research  and  Development  Cost  +  Procurement  Cost 


Figure  7:  Unit  Cost  Function 
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Table  5:  Results  of  Analyses 


Hypothesis  1 

(KP1) 

Hypotheses  2 

(KP2) 

Hypothesis  3 

(. KP3 ) 

SIP 

0.298 

0.468 

0.197 

QCP 

0.145* 

0.180* 

0.205* 

UCC 

0.054* 

0.066* 

0.049** 

TCC 

0.236 

0.172 

0.360 

RDCC 

0.001*** 

0.104 

0.390 

PCC 

0.339 

0.256 

0.294 

*  <  .1 
**  <.05 

***  <.01  fWelch  Test:  Unequal  Variances 

The  results  of  our  t-tests  indicate  that  if  a  program  manager  adheres  to  the  KAT 
they  will  reduce  their  unit  cost  growth.  As  mentioned  earlier  we  did  not  find  significance 
with  many  variables.  However,  we  believe  that  we  could  not  find  significance  with  PCC, 
TCC,  SIP,  and  QCP  because  of  the  inherent  variability  of  acquisition  programs,  with 
other  uncontrolled  factors  making  a  true  effect.  .  Nevertheless,  we  found  UCC  to  be  the 
most  important  variable  for  determining  better  program  performance.  Unit  Cost  changes 
when  any  of  the  other  variables  change.  Furthermore,  we  propose  that  UCC  is  a  better 
measurement  of  program  performance  because  of  this  transparency  and  because  it  can  be 
easily  compared  among  programs  of  different  types.  For  example,  a  satellite  program 
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may  have  a  large  R&D  cost  relative  to  its  Procurement  costs  while  a  dumb  munition  that 
is  mass-produced,  may  have  a  much  larger  Procurement  cost  relative  to  its  R&D  costs. 
This  can  make  R&D  or  Procurement  cost  growth  hard  to  compare  among  programs. 

Decreased  Variability 

In  most  cases,  the  cost,  schedule,  and  quantity  change  for  programs  that  met  the 
KAT  criteria  varied  less  than  for  programs  that  did  not.  Figure  8  through  Figure  10  show 
the  range  of  outcomes  for  each  variable  with  respect  to  the  Knowledge  Points  (0  = 

Missed  KP;  1  =  Met  KP).  In  many  cases,  programs  did  not  experience  better  results  by 
meeting  the  KP  criteria.  However,  some  programs  that  met  the  KP  criteria  appeared  to 
experience  decreased  variability.  A  double-asterisk,  in  the  following  figures,  indicates 
the  groups  with  statistically  different  variances  according  to  the  results  of  the  Levene  test 
for  equal  variance.  This  decrease  in  variability  may  be  significant  because  inspection  of 
the  data  seems  to  suggest  that  the  distribution  of  outcomes  has  a  heavy  tail  in  the  negative 
(or  “bad”)  direction.  Thus,  meeting  KP  criteria  might  reduce  the  probability  of  a  program 
performing  really  badly. 

Controlling  for  Program  Age 

Table  6  displays  the  results  of  our  alternative  model,  which  controlled  for 
program  age.  First,  we  found  that  the  relationship  between  RDCC  and  KP1  weakened. 
We  did  not  find  that  programs  that  missed  KP1  experienced  higher  Research  and 
Development  Costs  when  we  controlled  for  age.  However,  we  found  a  marginal 
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relationship  between  Research  and  Development  Cost  increases  and  programs  that 


missed  KP2.  While  we  cannot  explain  this  with  certainty  why  this  occurs,  we  do  note 
that  KP2  measurement  involves  a  near  complete  product  design,  and  hence  it  makes 
sense  that  programs  that  did  not  have  a  clear  understanding  of  their  product  as  they 
approached  production  experienced  more  cost  growth.  This  could  also  explain  why  KP1 
was  not  significant  after  we  controlled  for  age.  Milestone  B  could  be  too  early  in  the 
acquisition  cycle  to  be  a  reliable  estimate  of  R&D  cost  growth.  Programs  that  meet  KP1 
criteria  may  begin  with  an  advantage,  but  it  may  be  that  the  advantage  dissipates  with 
time  due  to  factors  exogenous  to  the  program  office. 


Table  6:  Results  of  Analysis,  Controlling  for  Age 


KP1 

KP2 

KP3 

SIP 

0.490 

0.239 

0.485 

QCP 

0.016** 

0.105** 

0.028** 

UCC 

0.003*** 

0.011** 

0.022** 

TCC 

0.233 

0.159 

0.384 

RDCC 

0.135 

0.096* 

0.282 

PCC 

0.337 

0.192 

0.289 

*  <.i 

**  <.05 
***  <.01 
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Second,  we  found  that  the  relationship  between  programs  that  met  the  KP  criteria 
and  lower  unit  cost  growth  increased  substantially.  Our  previous  tests  indicated  a 
marginal  relationship,  but  when  we  controlled  for  AGE  the  relationship  became 
significant  for  all  of  the  independent  variables.  Furthermore,  Quantity  change  became 
significant  with  respect  to  KP1  and  KP2.  In  other  words,  programs  that  met  KP  I  and 
KP2  experienced  a  smaller  reduction  in  quantities  than  programs  that  did  not  meet  these 
Knowledge  Points.  Naturally,  programs  have  more  opportunity  to  experience  cost  and 
schedule  changes  as  they  age.  By  controlling  for  age,  we  were  able  to  account  for  this 
artifact  and  get  a  clearer  understanding  of  how  the  KP  criteria  relate  to  unit  cost  and 
quantity  change.  In  other  words,  we  could  determine  if  the  cost  and  quantity  changes 
correlated  to  meeting  the  Knowledge-Points,  and  we  could  eliminate  the  possibility  that 
those  relationships  were  simply  a  function  of  a  program’s  age. 
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Figure  8:  KP1  versus  DV's 
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Figure  10:  KP3  versus  DV's 
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Next,  we  organized  the  programs  in  our  dataset  into  categories.  We  looked  to  see 
if  any  trends  existed  among  the  program  types.  Two  program  types,  Land  and  Space/Air, 
contained  only  one  data  point  so  we  could  not  obtain  any  trends  from  those  groups.  Of 
the  groups  we  could  assess,  we  found  that  Munition  programs  were  much  more  likely  to 
meet  KP1  criteria  than  any  other  program  type.  Additionally,  we  found  that  Space 
programs  and  Missile  programs  met  KP2  criteria  more  than  any  other  program  type.  We 
found  that  KP3  was  the  most  frequently  missed  KP;  only  7.8%  of  the  programs  assessed 
at  KP3  had  production  processes  in  statistical  control. 


■  Sum  of  Met  KP1 

■  Sum  of  Missed  KP1 

■  Sum  of  Met  KP2 

□  Sum  of  Missed  KP2 
IS  Sum  of  Met  KP3 
HSum  of  Missed  KP3 


Figure  11:  KP  accomplishment  by  program  type 


Lastly,  we  analyzed  whether  the  KAT  was  more  influential  on  one  program  type 
than  another  was  (Appendix  B).  We  expected  that  meeting  the  KP  criteria  would  have  a 
smaller  affect  on  certain  program  types  for  a  couple  of  reasons.  First,  the  outcomes  of 
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high  profile  programs  may  be  influenced  more  by  politically  driven  decisions  than  by 
program  execution  decisions.  Second,  simpler  programs  may  have  fewer  uncertainties 
remaining  after  KP  completion  providing  for  a  better  chance  at  improved  outcomes.  Our 
sample  size  was  reduced  significantly  when  we  categorized  our  variables  by  program 
type.  This  limited  the  conclusions  we  could  draw  from  the  data.  The  output  of  our 
analysis  is  in  Appendix  B.  Generally,  all  program  types  appeared  to  be  affected 
relatively  similarly.  However,  because  there  are  more  data  points  for  programs  missing 
the  KP  criteria  than  meeting  the  criteria  we  could  not  make  any  certain  claims.  The 
decreased  variability  and  better  outcomes  may  have  been  due  to  chance,  or  may  have 
been  an  artifact  of  the  smaller  sample  size. 

Lastly,  it  is  important  to  note  that  the  programs  that  met  the  KPs  had  certain 
characteristics  in  common.  For  instance,  not  many  Air  programs  met  the  Knowledge- 
Points.  However,  Air  programs  that  met  the  KPs  were  older  programs  that  were  being 
upgraded,  or  in  the  case  of  the  Light  Utility  Helicopter,  a  commercial-of-the-shelf 
product.  Furthermore,  we  found  that  the  preponderance  of  programs  that  met  the 
Knowledge-Points  were  in  either  the  missile  or  munition  category.  These  programs  are 
by  no  means  simple,  however  they  are  much  less  complicated  than  an  Airframe  or  Space 
system. 
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Chapter  V :  Conclusions 


The  GAO  has  championed  the  Knowledge-Based  Acquisition  Theory  for  years. 
Their  claim  has  been  that  when  programs  have  mature  technology,  a  complete  product 
design,  and  mature  production  processes,  they  will  experience  better  results.  Our  thesis 
focused  on  validating  the  GAO’s  claim  in  order  to  further  the  acquisition  community  of 
practice.  In  many  ways,  our  study  validated  the  GAO’s  claim.  However,  we  also  found 
that  the  KAT  is  not  a  comprehensive  means  to  program  success.  There  are  other  factors 
not  captured  in  the  KAT  that  contribute  to  program  success. 

We  could  not  find  a  significant  relationship  between  KP1,  KP2,  or  KP3,  and 
PCC,  SIP,  or  TCC  performance.  We  believe  that  this  is  because  acquisition  programs  are 
unique  and  will  not  all  be  affected  the  same  when  a  program  misses  KP1,  KP2,  or  KP3 
criteria.  However,  we  did  observe  that  when  a  program  missed  the  KP  criteria  its 
outcome  varied  much  more  wildly  than  programs  that  had  met  the  KP  criteria. 

Additionally,  we  observed  that  the  KAT  correlated  considerably  with  improved 
unit  cost  change  performance.  In  retrospect,  we  believe  that  the  UCC  variable  is  the  most 
suitable  measurement  of  success  of  all  the  variables  because  it  encompasses  several 
program  measurement  components.  Unit  cost  changes  in  any  program  when  either  a 
dollar  or  a  quantity  component  changes  (unless  cost  and  quantity  change 
proportionately).  Furthermore,  since  UCC  was  a  significant  variable  for  all  KP ’s  with  the 
alternative  model,  we  feel  its  correlation  validated  the  GAO’s  KAT  as  an  effective 
approach  for  better  program  outcomes. 
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Limitations 


Some  limitations  of  our  study  warrant  future  research.  First,  our  sample  size  was 
limited.  As  of  this  study,  the  GAO  had  only  been  tracking  KP  achievement  for  seven 
years  (2003  -  2009)  limiting  our  analysis  to  a  combined  107  programs,  and  some  of  these 
programs  had  only  partial  data.  To  date,  relatively  few  programs  have  met  the  KP 
criteria,  giving  us  unequal  sample  sizes.  More  programs  missed  the  KP  criteria  than  met 
them.  Additionally,  we  had  fewer  data  points  for  KP2  than  for  KP1,  similarly,  we  had 
fewer  data  points  for  KPS  than  for  KP2.  Our  dataset  only  contained  four  programs  that 
met  KP3  criteria  preventing  us  from  conducting  a  rigorous  analysis  of  the  correlation 
between  KP3  criteria  and  better  program  outcomes.  This  study  would  benefit  from  a 
retest  of  the  hypothesis  when  more  data  points  are  available. 

There  were  limitations  to  the  conclusions  we  made.  We  did  not  have  the  right 
kind  of  data  (e.g.  experimental)  to  test  for  causality.  Future  researchers  can  use  both  the 
data  provided  by  the  GAO  and  annual  program  data  from  the  Selected  Acquisition 
Reports  to  see  if  program  cost  and  schedule  performance  was  altered  once  a  program  met 
the  KP  criteria.  In  addition,  we  could  only  show  that  program  outcomes  were  in  general 
correlated  to  the  KP  criteria.  We  could  not  determine  if  certain  types  of  programs  were 
more  strongly  correlated  than  others.  A  larger  sample  of  programs  may  show  that  the 
KAT  is  only  applicable  to  specific  program  types. 

Lastly,  while  it  appears  that  programs  have  been  reluctant  to  adopt  the  GAO’s 
Knowledge-Based  Approach,  it  may  be  that  the  approach  must  be  adapted  to  the  DOD 
acquisition  environment.  Department  of  Defense  programs  are  by  nature  innovative  and 
cutting  edge,  thus  bearing  a  higher  degree  of  uncertainty  and  risk  than  commercial 
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programs  (Friedman,  2009).  The  DOD  program  managers  are  often  not  afforded  the  time 
and  resources  needed  to  obtain  the  same  knowledge  of  their  program  as  commercial 
program  managers.  It  may  be  unjust  to  compare  DOD  program  outcomes  to  the 
outcomes  of  Commercial  firms;  given  that  DOD  program  managers  are  expected  to 
manage  much  more  complicated  and  risky  programs.  However,  if  DOD  program 
outcomes  are  expected  to  measure  up  to  commercial  outcomes,  the  DOD  should  consider 
altering  its  practices  to  compare  to  the  best  commercial  practices  (e.g.  GAO’s  KAT). 

Impact  to  the  Acquisition  Community 

The  results  of  this  analysis  should  influence  the  decisions  made  by  the  acquisition 
community.  We  have  validated  that  adhering  to  the  GAO’s  KAT  is  an  effective  way  of 
improving  acquisition  results.  Furthermore,  we  can  interpret  that  adhering  to  the  KAT 
can  help  program’s  produce  their  products  more  closely  to  the  originally  estimated  cost. 
The  GAO’s  Knowledge -Based  Approach  can  help  program  managers  deliver  capabilities 
to  the  warfighter  more  closely  to  their  original  cost  and  schedule  estimates. 

Additionally,  the  GAO’s  findings  should  provide  program  managers  with  a 
substantial  reason  to  freeze  requirements  early,  and  to  use  mature  technology.  If  they  can 
avoid  requirements  creep  and  developing  technology  past  Milestone  B,  they  can  improve 
their  program  outcomes.  Ultimately,  the  warfighter  benefits  most,  because  they  will  get 
their  products  quicker  and  closer  to  the  original  cost  estimate. 
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Conclusion 


The  language  contained  in  the  Weapon  System  Reform  Act  of  2009  supports  the 
GAO’s  KAT  by  placing  emphasis  on  using  mature  technology  and  conducting  trade-offs 
between  cost,  schedule,  and  performance  (11 1-23,  2009).  We  validated  the  GAO’s  KAT 
as  a  useful  means  of  achieving  better  program  results,  but  we  were  unable  to  validate  it  as 
a  comprehensive  means  to  better  program  performance.  In  our  opinion,  the  Knowledge- 
Based  Acquisition  Theory  is  appropriate  for  DOD  programs;  it  can  help  them  experience 
better  results.  However,  we  also  acknowledge  that  in  some  cases  it  may  require  limiting 
a  product’s  capability  or  slowing  down  the  fielding  or  a  product  beyond  the  user’s 
tolerance  level.  For  instance.  Quick  Reaction  Capabilities  may  not  be  suited  for  the  KAT. 
Where  applicable,  the  GAO’s  Knowledge-Based  Acquisition  Theory  can  help  programs 
experience  better  results. 
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Appendix  A:  Technology  Readiness  Level’s 


(GAO,  2009,  pp 

174-175) 

Technology 
readiness  level 

Description 

H  ardwara/s  oftware 

Demonstration 

environment 

1 .  Basic  principles 
observed  and 
reported 

Lowest  level  ol  technology  readiness.  Scientific 
research  begins  to  be  translated  into  applied 
research  and  development.  Examples  might 
include  paper  studies  of  a  technology's  basic 
properties 

None  (paper  studies  and 
analysis) 

None 

2.  Technology  Invention  begins.  Once  basic  principles  are 

concept  andbr  observed,  practical  applications  can  be  invented 

application  formulated  The  application  is  speculative  and  there  is  no 
proof  or  detailed  analysis  to  support  the 
assumption.  Examples  are  still  limited  to  paper 
studies. 

None  (paper  studies  and 
analysis) 

None 

3.  Analytical  and 
experimental  critical 
function  andfor 
characteristic  proof  of 
concept 

Active  research  and  development  is  initiated-  This 
includes  analytical  studies  and  laboratory  studies 
to  physically  validate  analytical  predictions  of 
separate  elements  of  the  technology.  Examples 
include  components  that  are  not  yet  integrated  or 
representative. 

Analytical  studies  and 
demonstration  of  nonscale 
individual  components  (pieces  ot 
subsystem) 

Lab 

4.  Component  and/or 
breadboard  validation 
in  laboratory 
environment 

Basic  technological  components  are  integrated  to 
establish  that  the  pieces  will  work  together.  This 
is  relatively  low  fidelity'  compared  to  the  eventual 
system.  Examples  include  integration  of  ‘ad  hoc" 
hardware  in  a  laboratory. 

Low-fidelity  breadboard- 
integration  of  nonscale 
components  to  show  pieces  will 
work  together.  Not  fully 
functional  or  form  or  fit  but 
representative  of  technically 
feasible  approach  suitable  for 
flight  articles. 

Lab 

5.  Component  and/or 
breadboard  validation 
in  relevant 
environment 

Fidelity  of  breadboard  technology  increases 
significantly  The  basic  technological  components 
are  integrated  with  reasonably  realisbc  supporting 
elements  so  that  the  technology  car  be  tested  in  a 
simulated  environment.  Examples  include  “high 
fidelity’  laboratory  integration  of  components. 

High-fidelity  breadboard. 
Functionally  equivalent  but  not 
necessarily  form  and/or  fit  (size 
weight,  materials,  etc).  Should  be 
approaching  appropriate  scale. 
May  Include  integration  of  several 
components  with  reasonably 
realistic  support 
elements/subsystems  to 
demonstrate  functionality. 

Lab  demonstrating 
functionality  but  not  form 
and  fit.  May  include  flight 
demonstrating 
breadboard  in  surrogate 
aircraft.  Technology 
ready  tor  detailed  design 
studies 

6.  System/subsystem 
model  or  prototype 
demonstration  in  a 
relevant  environment 

Representative  model  or  prototype  system,  which 
is  well  beyond  the  breadboard  tested  for  TRL  5.  is 
tested  in  a  relevant  environment.  Represents  a 
major  step  up  in  a  technology's  demonstrated 
readiness  Examples  include  testing  a  prototype 
in  a  high  fidelity  laboratory  environment  or  in 
simulated  realistic  environment. 

Prototype.  Should  be  very  dose 
to  form,  fit  and  function.  Probably 
includes  the  integration  ot  many 
new  components  and  realistic 
supporting  elements/subsystems 
it  needed  to  demonstrate  full 
functionality  ot  the  subsystem. 

High-fidelity  lab 
demonstration  or 
limited/restricted  (light 
demonstration  for  a 
relevant  environment 

Integration  of  technology 
is  well  defined. 

7.  System  prototype 
demonstration  in  a 
realistic  environment 

Prototype  near  or  at  planned  operational  system. 
Represents  a  major  step  up  from  TRL  6,  requiring 
the  demonstration  of  an  actual  system  prototype 
in  a  realisbc  environment,  such  as  in  an  aircraft 
vehicle  or  space.  Examples  indude  testing  the 
prototype  in  a  test  bed  aircraft. 

Prototype.  Should  be  form,  fit  and 
function  integrated  with  other  key 
supporting  elements/subsystems 
to  demonstrate  full  functionality  of 
subsystem. 

Flight  demonstration  in 
representative  realistic 
environment  such  as 
flying  test  bed  or 
demonstrator  aircraft. 

Technology  is  well 
substantiated  with  test 
data. 

8.  Actual  system 
completed  and  flight 
qualified"  through  test 
and  demonstration 

Technology  has  been  proven  to  work  in  its  final 
form  and  under  expected  conditions.  In  almost  all 
cases,  this  TRL  represents  the  end  of  true  system 
darelopment  Examples  include  developmental 
lest  and  evaluation  of  the  system  in  its  intended 
weapon  system  to  determine  if  it  meets  design 
specifications 

Right-qualified  hardware 

Developmental  Test  and 

Evaluation  (DT&E)  in  the 
actual  system  application. 

9.  Actual  system 
flight  proven"  through 
successful  mission 
operations 

Actual  application  of  the  technology  in  its  final 
form  and  under  mission  conditions,  such  as  those 
encountered  in  operational  test  and  evaluation. 

In  almost  all  cases,  this  is  the  end  of  the  last  f>ug 
fixing"  aspects  of  true  system  development 
Examples  include  using  the  system  under 
operational  mission  conditions. 

Actual  system  in  final  form 

Operational  Test  and 

Evaluation  (OT&E)  in 
operational  mission 
conditions. 
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Appendix  B:  Summary  of  KP  achievement  by  Program  Type  and  Dependent 
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